Add vector_index virtual table - #3

Stacked on #2. Review that first; this PR's diff is only the commits below.

Summary

• Bump zombiezen.com/go/sqlite to v1.4.3-0.20260620165750-ca2041ed3dd2. This picks up Fix variable shadowing in resultError that silently swallows function errors zombiezen/go-sqlite#133, which fixes errors returned from SQL functions being silently dropped. The fix is merged but not yet tagged. Since v1.4.2 there are six commits: the fix, one docs change and four CI updates. The modernc versions are unchanged. This commit un-skips the 12 error tests blocked on the bug and removes an obsolete test that expected placeholder functions to error.
• Add the vector_index virtual table. It stores vectors in chunks of about 1 MiB inside shadow tables and answers k-nearest neighbor queries by scanning whole chunks in Go on GOMAXPROCS goroutines. It avoids a SQL function call for every row.

Usage

CREATE VIRTUAL TABLE docs_vec USING vector_index();    -- float32
CREATE VIRTUAL TABLE docs_q USING vector_index(int8);  -- requires WithQuantRange
CREATE VIRTUAL TABLE docs_c USING vector_index(float32, chunk_size=512);

INSERT INTO docs_vec (rowid, embedding) VALUES (?1, ?2);

SELECT rowid, distance FROM docs_vec WHERE embedding MATCH ?1 AND k = 10;
SELECT rowid, distance FROM docs_vec WHERE embedding MATCH ?1 ORDER BY distance LIMIT 10;

• k can be given as k = ? or as LIMIT, optionally with OFFSET.
• SQLite does not pass LIMIT through joins, so a join must use k or a subquery. A join that relies on LIMIT alone returns an error saying so.
• INSERT, UPDATE and DELETE are supported, including rowid changes and inserts without a rowid.
• Rename and drop move or remove the shadow tables.
• Reopening with a different dimension or quantization range than the table was created with is an error.

Chunk size

• The default is max(1, 1 MiB / vectorBytes) vectors per chunk. chunk_size=N overrides it. Both are stored when the table is created.
• A sweep over 128, 768 and 1536 dimensions compared chunk data sizes of 64 KiB, 256 KiB, 1 MiB and 4 MiB, for float32 and int8. Search was fastest near 1 MiB. Inserts were 4.5-16x slower at 4 MiB than at 1 MiB.
• The first version of this PR used a fixed 1024 vectors per chunk. That is 3 MiB at 768 dimensions and 6 MiB at 1536, which falls in the slow-insert range.
• The README has the full trade-off table.

Storage

• name_info: dimension, storage type, quantization range and chunk size.
• name_chunks: chunk id and occupied size. It is kept separate from the data because updating a row rewrites its blobs. The first version kept them together and took 5m44s to insert SIFT1M.
• name_data: one blob of rowids and one blob of vectors per chunk.
• name_rowids: maps each rowid to its chunk and slot.

A delete moves the chunk's last vector into the freed slot, so chunks never have holes.

Results

SIFT1M: 1M vectors × 128 dimensions, k=100, Apple M3 Max, p50.

• Recall matches the scalar functions.
• vector_index searches use all cores. The other rows are single-threaded.
• With mmap, most of the remaining search time is spent reading chunk blobs through modernc.

Known limitations

• At 128 dimensions, inserts are about 5x slower than into a plain table: each insert runs several statements and opens two blob handles. At 768 and 1536 dimensions, float32 inserts match a plain table, and int8 inserts are faster.
• At 128 dimensions, the old 1024-vector chunks (512 KiB) inserted SIFT1M in 9.8 s with the same search speed. 512 KiB was not included in the sweep.
• Reading the embedding column costs one lookup per row. Searches that return only rowid and distance do not pay this.
• The first search after a bulk load is slower, which inflates p99.

Testing

• go vet ./... and go vet -tags sift ./...
• go test -race ./...
• vector_index_test.go compares search results and distances against vector_distance / vector_distance_q brute force, for float32 and int8, across more than two chunks. It covers k, LIMIT, OFFSET, joins, deletes, updates, rowid changes, rollback, stored-embedding round trips, error cases (checking that failed statements leave rows intact), reopening, rename and drop, and default and explicit chunk sizes.
• go test -tags sift -run TestSIFT1M -timeout 0 -v -sift.dir .tmp/sift

🤖 Generated with Claude Code

https://claude.ai/code/session_01PTvMD95uUfGT1jasgFsLdP